Papers with OOD data
OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing models for OOD detection work with text, but they do not work directly with the text. |
| Approach: | They propose to use a sequential generative adversarial network (SeqGAN) based model to generate OOD data for a given domain automatically. |
| Outcome: | The proposed model outperforms state-of-the-art in OOD detection metrics for ROSTD and OSQ datasets. |
Bridging Distribution Gap via Semantic Rewriting with LLMs to Enhance OOD Robustness (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning on indistribution data fail to provide robustness against distribution shifts limiting the practical deployment of LLMs in dynamic real-world scenarios. |
| Approach: | They propose a method that leverages the flexibility of LLMs to align both in-distribution (ID) and OOD data with the LLM's distributions. |
| Outcome: | The proposed method outperforms fine-tuning methods on OOD tasks and benchmark datasets. |
Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models (2021.naacl-main)
Copied to clipboard
Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, Xia Hu
| Challenge: | Recent studies indicate that NLU models are prone to rely on shortcut features for prediction, without achieving true language understanding. |
| Approach: | They propose a shortcut mitigation framework to suppress NLU models from making overconfident predictions for samples with large shortcut degree. |
| Outcome: | The proposed framework suppresses the model from making overconfident predictions for samples with large shortcut degree. |
On Prefix-tuning for Lightweight Out-of-distribution Detection (2023.acl-long)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is a fundamental task vexing real-world applications . fine-tuning based methods require storing fine- tuned models for each scenario . |
| Approach: | They propose an unsupervised prefix-tuning based OOD detection framework called PTO . they propose to take advantage of optional training data labels and targeted OOD data . |
| Outcome: | The proposed framework performs better than existing methods under a wide range of metrics, detection settings, and OOD types. |
SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020.emnlp-main)
Copied to clipboard
| Challenge: | Data augmentation is a common method used to improve out-of-domain (OOD) generalization. |
| Approach: | They propose a data augmentation method that uses corruption and reconstruction functions to move randomly on a manifold to generate training examples. |
| Outcome: | The proposed method outperforms existing methods and baseline models on both in-domain and OOD data and achieves gains of 0.8% on OOD Amazon reviews, 1.8% accuracy on OOO MNLI, and 1.4 BLEU on in- domain IWSLT14 German-English. |
Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models suffer from severe miscalibration for both in-distribution and out-of-difference data due to over-parameterization. |
| Approach: | They propose a regularized method to improve in-distribution and out-of-distance calibrations by using on-manifold regularization and off-manfold regularisation. |
| Outcome: | The proposed method outperforms existing methods for text classification in terms of expectation calibration error, misclassification detection, and OOD detection on six datasets. |
Generalizing Few-Shot Named Entity Recognizers to Unseen Domains with Type-Related Features (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Few-shot named entity recognition methods struggle with out-of-domain (OOD) examples due to their reliance on manual labeling for the target domain. |
| Approach: | They propose a framework to enable generalization to an unseen target domain with only a few labeled examples. |
| Outcome: | The proposed framework achieves significant performance improvements on in-domain and cross-domain datasets. |
Teaching Small Language Models Reasoning through Counterfactual Distillation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance in a wide range of downstream tasks. |
| Approach: | They propose a counterfactual distillation framework that leverages LLMs to generate high-quality counterfacts and utilizes multi-view CoT to enhance the diversity of reasoning samples. |
| Outcome: | The proposed framework enhances reasoning capabilities of large language models and is more robust to OOD data. |
Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that biased samples can be brittle for VQA models . however, the improvements on OOD data severely sacrifice the performance on the in-distribution (ID) data. |
| Approach: | They propose a contrastive learning approach that exploits biased samples for unbiased information that contributes to reasoning. |
| Outcome: | The proposed method achieves competitive performance on the OOD dataset while maintaining robustness on the ID dataset. |
Types of Out-of-Distribution Texts and How to Detect Them (2021.emnlp-main)
Copied to clipboard
| Challenge: | Current NLP models produce unreliable or catastrophic predictions when training and test distributions differ . current models tend to produce unreliability or even catastrophic predictions that hurt user trust. |
| Approach: | They categorize examples as exhibiting a background shift or semantic shift and use calibration and density estimation methods to detect OOD examples. |
| Outcome: | The proposed methods beat calibration methods in background shift settings and perform worse in semantic shift settings. |
Privacy-Preserving Reasoning with Knowledge-Distilled Parametric Retrieval Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RAG systems require uploading local documents to the cloud, resulting in inference latency and poor generalization on out-of-distribution (OOD) inputs. |
| Approach: | They propose a generalizable knowledge-distilled parametric RAG model aligned with standard RAG in document structure and parameter activation. |
| Outcome: | The proposed model outperforms baselines in accuracy and generalizes well on out-of-distribution (OOD) data. |